Mathematical Models of Overparameterized Neural Networks

Deep learning has received considerable empirical successes in recent years. However, while many ad hoc tricks have been discovered by practitioners, until recently, there a lack of theoretical understanding for invented the deep literature. Known practitioners that overparameterized neural networks are easy to learn, past few years important developments analysis networks. In particular, it was shown such systems behave like convex under various restricted settings, as two-layer NNs, and when is locally so-called tangent kernel space around specialized initializations. This paper discusses some these progresses leading significant better We will focus on networks, explain key mathematical models, with their algorithmic implications. then discuss challenges current research directions.